UNCLASSIFIED 


ReoAoduced 
lui  iUe 


ARMED  SERVICES  TECHNICAL  INFORMATION  AGENCY 
ARLINGTON  HALL  STATION 
ARLINGTON  12,  VIRGINIA 


UNCLASSIFIED 


NOTICE:  When  government  or  other  drawings,  speci¬ 

fications  or  other  data  are  used  for  any  purpose 
other  than  in  connection  with  a  definitely  related 
government  procurement  operation,  the  U.  S. 
Government  thereby  incurs  no  responsibility,  nor  any 
obligation  whatsoever;  and  the  fact  that  the  Govern¬ 
ment  may  have  formulated,  furnished,  or  in  any  way 
supplied  the  said  drawings,  specifications,  or  other 
data  is  not  to  be  regarded  by  Implication  or  other¬ 
wise  as  in  any  manner  licensing  the  holder  or  any 
other  person  or  corporation,  or  conveying  any  rights 
or  permission  to  manufacture,  use  or  sell  any 
patented  invention  that  may  in  any  way  be  related 
thereto. 


i> 

^4viemorandum 

RM-3014-PR 

FEBRUARY  1962 


«dr 


LO  Y 

c- 

Li..;  .  '-’  I 

c-  "^5 


h:  ’-iCSl 

r  -■)  ' 


WHEN  TO  STOP  SAMPLING 
INITIATE  PRODUCT  IMPROVEMENT 


J.J.  McCall 


ASTIA' 


ijl  FFBS^‘^ 


r,s-'A  s 


PREPARED  FOR; 

UNITED  STATES  AIR  FORCE  PROJECT  RAND 


R*|-|  n  \) 

SANTA  MONICA  •  CAllfORMIA - 


MEMORANDUM 

RM-3014-PR 

FBBKUARY  1962 


WHEN  TO  STOP  SAMPLING 
AND  INITIATE  PRODUCT  IMPROVEMENT 

J.J.  McCaU 


This  research  is  sponsored  by  ihe  l^niled  Stales  Air  Force  under  Project  HAND -Con¬ 
tract  No  AF  W(638)-700- monitored  hy  the  Directorate  of  Development  Planning, 
Deputy  Chief  of  Staff,  Research  and  Technology,  Hq  liSAF.  Views  or  conclusions  con¬ 
tained  in  this  Memorandum  should  not  be  interpreted  as  representing  the  official  opinion 
or  policy  of  the  United  States  Air  Force.  Permission  to  quote  from  or  reproduce  portions 
of  this  Memorandum  must  he  obtained  from  the  HAND  Cor|)oration. 


R-ti  n 


1700  MAIN  ST,  •  SANTA  MONICA  •  CAllFOINIA 


RM-30ll^ 

iii 


PREFACE 

Ttiis  Memorandum  is  one  part  of  a  continuing  study  concerning  the  rela> 
tion  between  information  collection  and  system  effectiveness.  Kie  relation 
studied  here  is  that  between  information  collection  and  the  pieduct  improve¬ 
ment  activity.  This  topic  will  be  treated  more  completely  in  a  future 
Research  Memorandum-  This  Memorandum  is  being  distributed  now  to  a  limited 
audience  because  of  its  possible  relevance  to  current  Air  Force  interest 
and  actions  in  the  area  of  product  improvement. 


SUMMARY 
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This  Memorandum  considers  the  following  problem.  PHjnds  of  a  ceijtain 
amount  ai'e  made  available  for  the  improvement  of  a  particular  weapon  system. 
Hie  weapon  syatem  is  composed  of  several  subsystems  wh.ich  fall  independently. 
Sach  of  these  subsystems  is  an  investment  prospect  for  product  improvement. 

If  information  regarding  the  attractiveness  of  each  of  these  prospectts  were 
free,  cost  and  return  estimates  would  be  obtained  for  each  of  the  sutfsystems 
and  the  funds  would  be  allocated  on  the  basis  of  these  estimates.  Hbwever, 
such  information  is  frequently  costly.  When  this  is  the  case,  it  is  usually 
uneconomical  to  purchase  cost  and  benefit  estimates  for  each  subsystem. 

The  funds  expended  on  excessive  information  gathering  could  have  beeft  more 
profitably  invested  in  actual  product  improvement.  A  rule  is  required  which 
will  indicate  when  the  decision  maker  should  stop  gathering  information  and 
initiate  product  improvement.  The  purpose  of  this  Memorandum  is  to  jjirovide 


such  a  rule. 


RM-301^ 

vii 


CONTENTS 


PREFACE  .  iii 

SUMMARY  .  V 

Section 

I.  INTRODUCTION  . 1 

II.  A  STOPPING  RULE  WHEN  THERE  IS  NO  PRIOR  INFORMATION  OF 

COSTS  AND  RETURNS  .  3 

III.  A  STOPPING  RULE  WHEN  THERE  IS  SOME  PRIOR 

INFORMATION  OF  RETURNS  .  9 

IV.  CONCLUSION  .  13 

REFERENCES  . I5 


RM-3014 

1 


I .  INTRODUCTION* 

Product  improvement  is  a  term  used  by  the  Air  Force  to  describe  the 
process  by  which  resources  are  expended  to  enhance  weapon  system  performance. 
It  is  assxjmed  here  that  performance  is  improved  whenever  the  cost  of  pro¬ 
ducing  a  unit  of  weapon  system  readiness  is  reduced.  This  cost  includes 
the  downtime  costs  associated  with  the  various  repair  actions  as  well  as 
the  dollar  cost  of  these  actions.  These  improvements  are  assumed  to  occur 
within  the  given  management  system.  An  in-commission  or  ready  weapon  sys¬ 
tem  is  assumed  to  be  capable  of  performing  its  mission. 

The  following  problem  is  considered.  Funds  of  amount  K  are  made 

available  for  the  Improvement  of  a  particular  weapon  system,  llie  weapon 

system  is  composed  of  n  subsystems,  which  fail  independently.  Each  of 

these  subsystems  is  an  investment  prospect.  If  information  regarding  the 

attractiveness  of  each  of  these  prospects  were  free,  cost  and  return 

estimates  would  be  obtained  for  each  of  the  n  subsystems  and  the  K  funds 

** 

would  be  allocated  on  the  basis  of  these  estimates.  However,  such  infor¬ 
mation  is  frequently  costly.  When  this  is  the  case,  it  is  usxxally  uneconom¬ 
ical  to  purchase  cost  and  benefit  estimates  for  each  subsystem.  The  fiinds 
expended  on  excessive  information  gathering  could  have  been  more  profitably 
invested  in  actual  product  improvement.  A  rule  is  needed  which  will  tell 
the  decision-maker  when  to  stop  gathering  information  and  initiate  product 
improvement.  The  purpose  of  this  Memorandum  is  to  provide  such  a  mile. 

_ 

I  wish  to  thank  H.  Markowitz  and  B.  L.  Van  Horn  for  several  very 
helpful  suggestions. 

** 

For  a  solution  to  this  kind  of  allocation  problem  see  Ref.  1  and 

Ref.  2. 


r 
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II .  A  STOPPING  RULE  WHEN  THERE  IS  NO  PRIOR  IWPORMATION  OF  COSTS  AND  RETURNS 

Resources  of  aiaount  K  are  available  for  improving  a  particviiar  weapon 
system.  The  weapon  system  is  composed  of  n  subsystems.  A  maximum  of  kj^ 
of  the  K  resources  can  be  invested  in  the  i-th  subsystem  at  a  rate  of 

return, i  =  1,  2,  _ ,  n.  Any  amount  less  than  can  also  be  invested 

at  the  rate  of  return.  The  variables,  and/^,  are  assumed  to  be 

random  with  known  distributions.  The  random  variable,  kj^,  is  distributed 
by  ^]^(kj^)  and  the  random  variable,  is  independent  of  kj^  and  distri¬ 
buted  by  $2  known.  At  a  price  of  I  the 

decision  maker  can  purchase  information  on  the  exact  values  of  k^  and^^ 
for  any  i.  From  the  point  of  view  of  and  I  can  be  regarded  as 

the  cost  of  drawing  a  sample  of  size  one  from  each  distribution.  In  terms 
of  the  Air  Force  product  improvement  program,  I  can  be  interpreted  as  the 
cost  of  obtaining  engineering  estimates  of  the  improvement  that  can  be 
anticipated  for  the  i-th  subsystem  and  the  associated  cost.  Such  estimates 
are  a  necessary  prelude  to  any  improvement  project.  Since  it  is  assimed 
that  no  prior  information  is  available  on/<?and  kj^  for  any  subsystem,  the 
decision  maker  is  indifferent  to  whether  information  is  purchased  first 
for  subsystem  i  and  then  possibly,  if  the  stopping  rule  permits,  for  sub¬ 
system  J  or  first  for  subsystem  J  and  then  possibly  for  subsystem  i.  The 
next  section  considers  the  case  where  there  is  some  prior  information  on 
returns  and,  therefore,  a  prior  ranking  of  candidates. 

Suppose  that  the  K  funds  could  be  invested  at  a  rate  of  ret\irn, 
in  some  project  ,Pq^  other  than  the  improvement  of  the  weapon  system  under 
consideration.  Without  incurring  any  informational  cost,  a  return  of 
/^K  can  be  had  with  certainty.  The  alternative  is  to  purchase  information  on 


J 


RM-30lli 

h 


^  and  ki  for  the  i-th  subsystem-  Bie  cost  of  this  information  is  I-  It 
is  assumed  that  after  each  observation,  say  the  m-th,  on  and  kj^,  the 
decision  maker  can  decide  vhether  to  purchase  another  observation  for 
I  dollars  or  to  stop  sampling  suid  invest  the  K-rtI  funds  in  the  known  in¬ 
vestment  prospects.*  The  following  rule,  exemplified  for  the  first  stage 
decision,  irflll  be  used  to  decide  between  these  two  alternatives: 

If  the  expected  return  from  the  investment  opportunities  available 
after  spending  I  exceeds /’qK,  pvirchase  information  on and  kj^. 

If  the  expected  return  from  the  investment  opportunities  available 
after  spending  I  is  less  than  invest  in  P^. 

The  expected  return,  E(R),  after  spending  I  is: 


♦Problems  similar  to  this  have  received  considerable  attention. 

J.  L.  Fisher  in  Ref.  k  obtains  optimal  decision  rules  for  the  following 
sltviations:  A  decision  maker  is  faced  with  various  investment  opportimi- 
ties  over  time,  and  ^en  confronted  with  a  given  investment  opportunity 
must  accept  or  reject  it— a  rejected  opportunity  is  lost  forever.  The 
decision  maker  is  assxmed  to  have  a  fixed  budget  which  can  be  invested 
over  a  given  time  period.  He  is  not  permitted  to  accumulate  investment 
opportvinities  and  postpone  his  investment  decision.  The  optimal  decision 
rules  specify  vrtiich  Investments  should  be  accepted  and  which  shotild  be 
rejected. 

Some  work  has  also  been  done.  Refs.  U  and  5,  on  the  problem  of  when 
to  stop  sampling  and  accept  a  specified  payoff.  In  particular  the  follow¬ 
ing  problems  have  been  solved:  Let  CxJ  be  independent  random  variables 
with  a  common  distribution  function  F(x).  The  Xjj  are  observed  sequentially, 
and  if  sampling  is  terminated  at  Xjj,  a  payoff  of  either 

fn(Xi—  Xn)  =  X^  -  Cn  (Ref.  4) 


or 


fn(Xi  —  Xn)  =  Max  X^  -  Cn  (Ref.  5) 


is  acquired  where  C>0  is  the  cost  per  observation. 
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E(R)  *  /^(K- 


-i)J  yA 

/>o  K-I 
<vo  K-I 


(K 


If is  between  zero  and^,  the  first  integral,  then  regardless 
of  the  value  of  k^,  is  preferred  to  this  investment  opportvinity. 
However,  only  K-I  funds  can  be  invested  in  after  information  is  pur¬ 
chased. 

If^^  exceed6/<3^  and  k^  exceeds  K-I,  the  second  integral,  investing 
in  subsystem  i  is  preferred  to  and  K-I  can  be  invested  in  sybsystem  i 
at  an  expected  rate  of  return,  S(^^\yi>^;::yg) . 

If  /o^  exceeds and  is  between  zero  euid  K-I,  the  third  integral, 
E(k^j 0<k^<K-l)  could  be  Invested  in  subsystem  i  at  an  expected  rate  of 
return,  EC/'i  remainder  of  K-I,  K-I-E(kj^(04kj^<.K-l)  can  be 

invested  in  at  /’q. 

At  each  stage  of  the  process  a  similar  expectation  can  be  computed. 
Tlie  expected  return  of  another  piece  of  information  after  m  piirchases 
have  been  made  will  depend  upon  the  known  valiaes  of  k^  and^^,i=l , . .  .m, 
as  well  as K  and  I.  Given  these  values  the  computation  of  the  ex¬ 
pected  return  is  straightforward.  This  expectation  is  compared  with  the 
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best  return  that  can  be  made  from  known  investment  oppoirtxinitieB.  If 
this  expectation  exceeds  this  best  return,  another  piece  of  information 
is  purchased.  If  the  expected  return  is  less  than  the  best  return  aiready 
available,  the  sampling  process  is  terminated  and  the  remaining  prodluct 
improvement  funds  are  invested  in  the  best  of  the  known  prospects. 

This  stopping  rule  is  sensible  only  if  the  expected  gain  fixjm  lan 
additional  pvurchase  of  information  declines  as  more  information  is  pur¬ 
chased.  If  this  were  not  true,  then  it  might  pay  to  take  two  or  more 
additional  observations  even  though  the  cost  of  one  more  observation 
exceeded  its  expected  return.  A  stage  by  stage  decision  rule  wo\ild  be 
invalid.  In  the  process  described  here,  it  is  trxie  that  the  marginal 
valvie  of  information  is  a  non-increasing  function  of  the  number  of  ob¬ 
servations.  Ihis  is  illustrated  in  Fig.  1  where  the  Investment  oppor¬ 
tunity  curves  are  shown  for  various  sample  slses.  The  investments  are 
ranked  in  the  order  of  attractiveness.  For  example,  in  curve  n=>3,  the 
most  favorable  investment  is  k-k^^,  and  the  next  best  is  k:sk2^ 

and  the  least  favorable  is  k=K-k2^-k2  and  *  k-ki-'k2* 

that  the  cost  of  information  is  zero,  as  the  sample  size  Increases  the 
investment  opportunity  curve  moves  upward,  the  total  return  to  the  K 
resources  Increases.  For  example,  the  total  return  that  can  be  obtained 
from  the  K  funds  is  after  n^^  observations  and  after  n^  observations.* 
Consider  an  Investment  opportunity  which  is  the  n^^+lst  observation.  Sup¬ 
pose  that  by  investing  in  this  opportunity  the  n^^  curve  evaluated  at  K 
is  raised  from  Rj^  to  r|^.  For  any  investment  opportunity  curve  with 
n>n2^,  the  value  of  this  nj^+lst  observation  can  be  no  greater  than  its 

*For  purposes  of  exposition,  these  cxirves  are  assumed  to  be  continuous. 
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value  to  the  portfolio.  This  follows  from  the  fact  that  any  opportunity 
available  to  the  n,  portfolio  will  be  available  to  the  portfolio. 

•4.  C. 

Therefore,  the  investment  which  replaces  in  n^  portfolio  can  be  no 

more  valuable  than  the  investment  which  it  replaces  in  the  n^^  portfolio. 

But  this  is  true  of  every  possible  investment  opportunity  which  could 
be  revealed  by  n^^+lst  observation;  its  value  at  n^  must  be  no  less 
than  its  value  at  n^.  Therefore,  the  expected  return  from  an  additional 
observation  at  must  be  greater  or  equal  to  its  expected  return  at  02* 

A  fortiori,  this  is  true  if  the  cost,  I,  of  each  obsejrvation  is  greater 
than  zero;  i.e.,  if  the  cost  of  an  additional  observation  exceeds  its 
expected  return  at  n^,the  cost  of  an  additional  observation  at  n^  will 
exceed  its  expected  return. 


Fig  1 
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III.  A  STOPPING  RULE  WHEN  THEIRfi  IS  SOME  PRIOR  INFORMATION  OF  RETURMS 
The  case  is  now  considered  where  the  n  subsystems  can  be  ranked 
by  the  amount  of  mischief  each  is  causing,  where  mischief  is  measured  as 
follows.  Let  be  the  exponential  failure  rate  of  the  ito  subsystem. 
Whenever  the  1^  subsystem  fails  the  weapon  system  is  out  of  commission 
for  periods  and  a  financial  maintenance  charge  of  is  incurred. 
Assuming  that  the  subsystems  fail  independently,  the  weapon  system 
failure  rate.  A,  is  the  sum  of  the  individual  subsystem  failure  rates. 

The  mean  time-between-failures,  Af,  of  the  weapon  system  is 

The  expected  weapon  system  downtime,  D,  at  each  failure  is 

L-  I 

The  expected  maintenance  charge,  C,  at  each  failure  is 

c  =>• 

i=l 

The  weapon  system  amortization  rate,®f,  is 


o(  = 


H 


f 


where  is  the  initial  cost  of  the  weapon  system  and  H  is  its  expected 


service  life 
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It  follows  that  the  ratio,  M,  of  expected  cost,  L,  during  a  cycle 
to  expected  good  time,  T,  during  a  cycle,  the  cost  per  unit  of  weapon 
system  uptime,  is 


where 


and 


M=^^ 

T 


L  =  o<r(A+D)  +  C 


T  =/i 


Therefore,  M  can  be  written  as 


n 


M  =  oc+  oc  ^  ^  D  +  V 

i^  ^  ^  hr  ^  ^ 


The  higher  M,  the  less  satisfactory  is  weapon  system  performance. 
The  contribution,  S^,  of  the  ith  subsystem  to  M  is  simply 

Si  =  ^  ^i'^i 

or  if  there  are  di  subsystems  of  type  i,  then  the  contribution.  Si,  of 
the  ith  subsystem  is 


S  =  d  S 
i  i  i 

The  value  of  S^  measures  the  cost  per  unit  of  weapon  system  good 
time  which  the  ito  subsystem  is  imposing  on  the  weapon  system.  It  also 
measures  the  maximum  return  obtainable  from  improvement  expenditures  on 
the  ith  subsystem.  This  maximum  return  would  occur  if  were  reduced 
to  zero  or  if  both  Di  and  Ci  were  reduced  to  zero. 
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As  an  example,  suppose  that  a  weapon  system  is  composed  of  four  sub¬ 
systems,  one  of  type  1,  one  type  2, and  two  of  type  3.  Bxe  failure  rates 
for  the  three  different  subsystems  are,  respectively,  =  12  failvires 
per  year,  =  ih  failures  per  year  and  =  10  failures  per  year;  the 

initial  cost  of  the  weapon  system  is,  K  =  $1  million  and  its  expected 

Kn 

service  life  is  H  =  5  years,  so  that  o<r=  —  =  $200,000  per  year;  the  down- 

n 

times  incurred  because  of  a  failure  of  subsystem  1,  subsystem  2,  or  sub¬ 
system  3  are>  respectively,  =  .001  years,  =  .002  years, and  =  .004 
years;  finally,  the  maintenance  charges  resulting  from  a  failure  of  sub¬ 
system  1,  subsystem  2,  or  subsystem  3  are,  respectively,  =  $1000, 

Cg  =  $800,  and  =  $700.  Uhder  these  conditions,  the  cost  per  imit  of 
weapon  system  good  time  which  each  of  the  subsystems  (l,  2,  and  3)  imposes 
on  the  weapon  system  are,  respectively, 

+ 


=  d^  (oc  4  C^)  =  $14,400 

So  =  *  $1-6,800 

^  t  ^2  2  2 


and 


=  d 


(oc.  4  C^)  =  $30,000 


In  terms  of  the  previous  section,  if  k.  is  distributed  as  before, 

.  ^  S+ 

then  the  distribution  of  will  be  truncated  at  .  Given  k. ,  the 

i  1 

maximum  rate  of  return  from  these  k^  funds  is  the  maximum  return  divided 
by  k^.  Assuming  that  the  distribution  of  />  ^  is  the  same  as  before  except 
for  this  truncation,  then  given  k^,  the  larger  S^,  the  greater  is  the 
expected  value  of  —  the  mean  of  a  distribution  truncated  at  f  increases 
with  /  .  For  this  reason  it  is  now  advantageous  to  order  purchases  of 
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information  according  to  S^.*  That  subsystem  with  the  highest  should 
be  investigated  first;  the  subsystem  with  the  second  highest  value  of 
should  be  investigated  next,  etc.,  until  the  decision  maker  stops  purchasing 
information. 

In  terms  of  the  previous  example,  =  $l4,h00,  =  $l6,800,  and 

s'*"  =  $30,000, which  implies  that  subsystem  3  should  be  investigated  first, 

3 

then  subsystem  2  and  finally  subsystem  1.  As  before,  the  decision  maker 
stops  collecting  information  when  the  expected  return  after  spending  another 
I  on  information  is  less  than  the  return  available  prior  to  this  purchase. 


The  Air  Force,  realizing  that  it  would  be  imeconomical  to  gather 
information  on  every  subsystem,  has  developed  two  methods  for  isolating 
product  improvement  candidates.  The  first  relies  solely  on  the  behavior 
of  the  subsystem  failure  rate  over  time.  A  subsystem  whose  failure  rate 
is  out  of  conti*ol  with  respect  to  some  standard  failure  rate  is  automatically 
a  candidate  for  product  improvement;  i.e.,  kj^  and  /?.  are  estimated  for  out- 
of -control  subsystems.  The  second  method  isolates  those  subsystems  which 
are  the  highest  manhour  consumers  —  the  subsystem  whose  replacement  and 
repair  requires  the  highest  labor  expenditure  in  manhours  is  the  most 
lucrative  candidate  for  product  improvement,  etc.  Neither  method  considers 
downtime,  number  of  similar  subsystems  in  weapon  system  or  the  amo.  ization 
cost.  In  addition  the  first  neglects  maintenance  cost  while  the  second 
neglects  failure  rate. 
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IV.  CONCLUSION 

This  Memorandvm  has  considered  the  relation  between  information  col¬ 
lection  and  product  improvement.  In  particular,  the  following  problem  was 
considered.  Funds  of  amount  K  are  made  available  foiH  the  impiovement  of  a 
designated  weapon  system.  The  weapon  system  is  composed  of  n  subsystems 
which  fail  independently.  Each  of  these  subsystems  Us  an  investment  pros¬ 
pect.  If  information  regarding  the  attractiveness  of  each  of  these  prospects 
were  free,  cost  and  return  estimates  would  be  obtained  for  each  of  the  n 
subsystems  and  the  K  funds  would  be  allocated  on  the  basis  of  these  estimates. 
However,  such  information  is  frequently  costly.  Wher^  this  is  the  case ,  it 
is  usually  uneconomical  to  purchase  cost  and  benefit  lestimates  for  each 
subsystem.  The  funds  expended  on  excessive  informati|on  gathering  coviid 
have  been  more  profitably  invested  in  actual  product  improvement.  Iliis 
Memorandum  has  developed  a  rule  whereby  the  decieion  maker  can  determine 
when  to  stop  gathering  information  and  initiate  prodi^ct  improvement. 

In  the  course  of  developing  this  inile,  a  simple  technique  was  suggested 
for  ranking  product  Improvement  candidates  ( subsystei^s ) .  Ihe  ranking  of 
a  particular  subsystem  depended  on  (l)  the  subsystem's  failure  rate,  (2) 
the  number  of  subsystems  of  this  type  in  the  weapon  System,  (3)  the  cost 
in  weapon  system  downtime  of  a  subsystem  failure,  (4)  the  maintenance  cost 
of  a  siibsystem  failure,  (5)  the  initial  cost  of  the  \4eapon  system  and  (6) 
the  weapon  system's  expected  service  life. 
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